Updated the test refs, but I'm not sure they're right. I'm getting lots of diffs for a lot of tests. Running pylint built from main on Windows 11 (astroid-2.12.13 pylint-2.16.0.dev0).

It appears I'm getting the correct output with Ubuntu 20.04, but python test_functional.py --update-functional-output is not writing any changes even though this test fails still. 🤔

Should be good now, let me know if there's any issues @DanielNoord! I had to add a print into the tester to get the results :/

@DanielNoord is there anything else needing to be done?

@DanielNoord is there anything else needing to be done?

The CI fails so we would first need to fix that before this can be merged.

For whatever reason, both Python 3.8 and 3.10 do not work to update the abstract method file. I'm running python test_functional.py --update-functional-tests, but it just fails and doesn't update the file (it updates other files that we don't want).

-e git+https://github.com/sshane/pylint@c8564bb99a393621341851acaa3ed042b3c53c3d#egg=pylint
astroid==2.12.14
Python 3.10.0
Ubuntu 20.04

self = <pylint.testutils.functional.lint_module_output_update.LintModuleOutputUpdate object at 0x7fa1e3c6d690>

    def runTest(self) -> None:
>       self._runTest()
E       AssertionError: Wrong results for file "abstract_method":
E       
E       Unexpected in testdata:
E         47: abstract-method

../pylint/testutils/lint_module_test.py:145: AssertionError
=========================================================================================================================================== short test summary info ===========================================================================================================================================
FAILED test_functional.py::test_functional[abstract_method] - AssertionError: Wrong results for file "abstract_method":
============================================================================================================================ 1 failed, 779 passed, 25 skipped in 62.76s (0:01:02) =============================================================================================================================

Could it be that the test simply fails? If haven't checked that myself locally

Do you have any docs for what the comments do? For example, I'm not sure if # [abstract-method] means we're asserting an abstract method error from pylint, or we're catching an error to not fail the tests.

Are the functional tests a sort of blend of replay with the file outputs and unit tests?

Do you have any docs for what the comments do? For example, I'm not sure if # [abstract-method] means we're asserting an abstract method error from pylint, or we're catching an error to not fail the tests.

Are the functional tests a sort of blend of replay with the file outputs and unit tests?

https://pylint.readthedocs.io/en/latest/development_guide/contributor_guide/tests/writing_test.html#functional-tests

This (attempts to) describe the functional test framework. Basically it tries to lint the file and sees where we expect messages to be raised and compares that against the actual output of the linting.

@sshane This is quite close to being done. Do you have intention of working on it? Or shall we close it?

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.22%. Comparing base (3d7ac12) to head (9021d25).
⚠️ Report is 1 commits behind head on main.

@@           Coverage Diff           @@
##             main    #7955   +/-   ##
=======================================
  Coverage   96.22%   96.22%           
=======================================
  Files         178      178           
  Lines       19699    19700    +1     
=======================================
+ Hits        18956    18957    +1     
  Misses        743      743           

@DanielNoord I'm having the same problem I did when I last worked on it, I verified my # [abstract-method] markers are accurate according to running pylint on the file, but still get some test failures:

    def runTest(self) -> None:
>       self._runTest()
E       AssertionError: Wrong message(s) raised for "abstract_method.py":
E       
E       Unexpected in testdata:
E         47: abstract-method

Running python tests/test_functional.py --update-functional-output -k "test_functional[abstract_method]" does not update abstract_method.txt either

Narrowed it down to #9688. I don't have any more time to spend on this for the time being, feel free to get it across the line if it's simple for you, else you can close.

I was expecting each line's warnings to be compared to the annotations, not one by one! A more clear error message would have been nice 🙂

@DanielNoord I added the count which would have surfaced this for me much quicker, let me know if it's something you want and I can split it out

    def runTest(self) -> None:
>       self._runTest()
E       AssertionError: Wrong message(s) raised for "abstract_method.py":
E       
E       Unexpected in testdata:
E         47: abstract-method (2)

🤖 Effect of this PR on checked open source code: 🤖

Effect on astropy:

The following messages are now emitted:

Effect on pytest:

The following messages are now emitted:

Effect on sentry:

The following messages are now emitted:

Effect on black:

The following messages are now emitted:

Effect on home-assistant:

The following messages are now emitted:

This comment was truncated because GitHub allows only 65536 characters in a comment.

This comment was generated for commit 9021d25

As incredible as it seems, every new primer warning I checked seem to be an actual issue.